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ABSTRACT 


Human judgement and the process of decision making have been 
studied in depth for the past century. More recent research has 
revealed that feedback is a primary element in the decision making 
process. Feedback has been categorized according to its role in 
decision making. Some categories of feedback include cognitive, 
feedforward, and outcome. Cognitive feedback may hold the most 
promise for positively affecting the decision making process. 

Naval Intelligence analysis is a complex process which involves 
human judgement and decision making on a daily basis. This thesis 
sought to determine that cognitive feedback would enable 
intelligence analysts to make optimal choices more consistently 
than if they were presented with just outcome feedback. Naval 
Intelligence analysts were the subjects of an unclassified 
experiment which captured a realistic task performed routinely by 
analysts in the Fleet. The experiment revealed that cognitive 
feedback produced the most accurate and optimal results, and 
indicates that intelligence analysis would benefit from decision 
support systems that incorporate the element of cognitive feedback. 
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INTRODUCTION 


I . 

A. BACKGROUND 

Research on decision making has revealed the role of 
feedback in affecting the decisions of individuals (Hogarth, 
1987), Feedback affects the choice that a decision maker will 
make, and can be a crucial element in the decision making 
process, as it affects both the task environment and the 
decision maker’s perception of the environment (Hogarth, 
1987). 

Feedback can be categorized into several different types, 
including cognitive and outcome feedback. As described by 
Jacoby, et al, 1984, 

...outcome feedback is information that describes the 
accuracy or correctness of the response, cognitive 
feedback represents information regarding the how and why 
that underlies this accuracy. 

A type of feedback which has shown beneficial results is 
cognitive feedback (Balzer et al, 1989). Cognitive feedback 
may have particular utility in a complex, probabilistic 
environment, where pertinent information, provided to the 
decision maker in a timely fashion, can lead to improved 
decisions. An example of such an environment is military 
intelligence analysis. The examination of the effects of 
cognitive feedback on intelligence analysts may lend 
validation to current beliefs about cognitive feedback, and 
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raise additional questions which, when answered, could provide 
a basis for future intelligence decision support systems. 

B. EXPERIMENTAL TASK 

To best capture the effects of cognitive feedback on 
intelligence analysis, a task specifically designed to capture 
the analysis process of analysts in the Navy was devised. The 
task was presented to active duty Naval Intelligence analysts 
in a format similar to what is used in the Fleet. The entire 
experiment was unclassified, and the geopolitical scenario was 
fictitious, yet effort was taken to design a realistic task 
environment that the analysts would be familiar with. 

C. RESEARCH QUESTION 

Previous studies of military intelligence analysts have 
revealed that analysts will tend to seek confirmation of their 
decisions (Tolcott et al, 1989). If this confirmation can be 
provided in the form of cognitive feedback, will the analysts 
use the feedback to reach more correct and optimal decisions? 
Does the form of feedback matter, or will there be a 
noticeable variance between subjects provided with and without 
cognitive feedback? Answers to these questions were sought 
through the design and implementation of the experiment, and 
were the focus of this research. 

Analysts were randomly divided into two groups: cognitive 
feedback and outcome feedback. Each group was presented with 
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the same exact task, but the information presented as feedback 
varied, dependent upon which group the analyst was assigned 
to. The results of each group were compared to one another, 
and the effects of different types of feedback on the analysts 
decisions were collected. 

D. CONTRIBUTION 

The importance of the decision making process to 
intelligence analysis is recognized, and the need to determine 
which types of feedback are optimal has been previously stated 
(Thompson et al, 1984). This research attempted to further 
determine if cognitive feedback is relevant and beneficial to 
the process of intelligence analysis. 

Chapter II provides a literature review of research in the 
area of intelligence analysis, and lays the theoretical 
premise for the study. Chapter III describes in detail the 
experimental method. Chapters IV and V discuss statistical 
results of the experiment and conclusions drawn from the 
experiment, respectively. Results of this study and any 
follow-on studies may be used to develop more advanced 
decision support system prototypes for the Naval Intelligence 
and other military intelligence communities. 


V 
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II. THEORETICAL PREMISE 


A. THE DECISION MAKING PROCESS 
1. Complexity 

Human decision making is a complex process. Studies 
of this process have revealed that the complexity of decision 
making is dependent upon many individual factors such as 
intuition, anticipatory tendencies, and judgemental heuristics 
(Hogarth, 1987). A large portion of the decision making 
process involves human memory and the capability, or lack 
thereof, of the human mind to process information (Hogarth 
1979). 

The human being is a fallible decision maker (Hogarth 
1979). The human mind is unable to effectively process 
continuous information from a multitude of sources, especially 
in a stressful environment. A stressful environment places 
the human mind in a position to ignore some information and 
process other information, dependent upon existing heuristics 
(Hogarth, 1987). In applications where decision making is 
vital, such as diagnostic medicine and tactical military 
scenarios, it is imperative that the information assigned the 
least importance is indeed worthy of such a low value. 

Each piece of pertinent information contributing to a 
decision must be appropriately presented to the decision maker 
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in a manner that will ensure the information is processed with 
due value. It is beneficial to fully understand the 
importance of information presentation in the decision making 
process. This understanding can be accomplished by studying 
various decision models as described in subsection II.A.2. 

2. Decision Making Models 

Researchers have attempted to capture the human 
decision making process through observation and mathematical 
modeling. Two models of decision making are the conceptual 
judgement model (Hogarth, 1987) and the lens model, as 
described by Brunswick (Libby, 1981). The conceptual 
judgement model and the lens model appear to approach the task 
of decision making from the different aspects of psychology 
and mathematics. Yet the two models are inextricably linked 
by the element of feedback, a vital part of the decision 
making process. Section II.C identifies and defines feedback 
and its varieties in more detail. By examining each of these 
models a point of departure for this study was attained. 

a. Judgement Model 

Hogarth's conceptual judgement model divides the 
process of decision making into seven basic steps: (1) task 
environment, (2) Schema, (3) Acquisition, (4) Processing, (5) 
Output, (6) Action, and (7) Outcome. Step (7), outcome, 
provides information that will be pertinent to future 
decisions. This outcome of the decision making process is 
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also considered a form of feedback for future judgmental 
scenarios. Hence outcome, or feedback, is a means by which 
future judgements can be affected. In becoming pertinent 
information to the decision making process, feedback 
establishes its vital role in the decision making process. 

The conceptual model of human judgement is 
depicted in Figure 1. Feedback plays a crucial role in the 
judgement process, as it provides criteria to both the task 
environment and the judgement schema. The lens model as 
interpreted by Brunswick continues the description of the 
element of feedback in mathematical detail. 

b. The Brunswick Lens Model 

The Brunswick Lens Model allows the environment to 
be observed through a "lens of imperfect cues" (Libby, 1981), 
and assigns weights to each of the cues in a specific decision 
making scenario. The Lens Model is readily adapted to 
situations in which the decision making occurs in a 
probabilistic environment, and the accuracy of a decision is 
contingent both on the individual decision maker and the 
environment in which the decision is made (Hogarth, 1987). 

The lens model, as depicted in Figure 2, 
represents the differences between actual value weights 
assigned to pieces of pertinent information by the 
environment, and the value weights assigned to the pertinent 
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pieces of information by the decision maker. These pieces of 
information which are considered pertinent to a specific 
decision, or task, are formally referred to as cues. 



Figure 2. Brunswick Lens Model (c.f. Dudycha and Naylor 
1966; Libby 1981). 


The lens model uses identical polynomial equations 
for each side of the model; the environment and the decision 
maker. The differences lie in the assigned values, or 
weights, for each of the cues. All of the variables used by 
the lens model are listed and defined in Table 1. 
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TABLE 1. LENS MODEL VMtIABLES (c.f. Libby, 1981) 


Symbol 

Name 

Definition 

Re 

Environmental 

Predictability 

Rfele' 

R, 

Predictability 

RtsTs' 

G 

Matching Index 

^To'Ts' 


Achievement 



First, the lens model attempts to capture the 
status of the environment. The environment is represented by 
equation 1. 

y^=a+jbiXi+i)2X2+. . 


( 1 ) 

Yg, the actual model imposed by the environment, is a 
compilation of the weights (b2 ....bjj) assigned to the cues 
by the environment. Likewise, the decision maker's 
view of the task is captured in an identical equation, Yj, 
with the weights (b^. . . .h^) assigned to the cues (xp..Xjj) by 
the decision maker. 

Both the environment and the decision maker are 
using the same set of cues, (x^...Xjj), to which weights are 
then assigned. In the most ideal situation, the decision 
maker employs consistent weights for specific cues, dependent 
upon the nature of the task. 
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other factors that affect the decision-making 
process and are thus represented in the lens model include 
consistency and environmental noise. Environmental noise is 
also referred to as predictability (Libby, 1981). Both 
consistency and predictability utilize similar variables in 
their calculations. 

Rj, consistency, measures how consistent the 
decision maker is in predicting the environment. It is 
calculated by a multiple correlation between the cues and the 
decision maker’s judgements. 

Rg, predictability, measures how predictable the 
environment is. If noise or predictability is low, the 
probability of the decision maker attaining a reasonable task 
accuracy is low (Libby, 1981). Predictability can be 
calculated by employing a multiple correlation of the cues and 
the actual environmental values. 

The accuracy of decisions can be measured through 
the lens model, in the form of decision achievement, r^. 

r^=GR^s 


( 2 ) 

Consistency, R^, and predictability, Rg, can be calculated to 
determine the overall accuracy of the decisions made by the 
decision maker. Decision achievement is an indication of how 
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well an individual, Yj, matches the environment, , through 
this "lens of imperfect cues" (Libby, 1981). 

The Brunswik Lens Model lends well to the process 
executed by Naval Intelligence and Cryptological analysts. By 
examining the analyst's decision and mathematically comparing 
it to the environment's actual outcome, an index of 
achievement for intelligence analysts can be arrived at for a 
specific type of analysis task. The effects of various 
factors, such as feedback, on the decisionmaking process, can 
be examined by comparative analysis of achievement indices for 
multiple instances of a generic task. 

B. INTELLIGENCE ANALYSIS AS A DECISION MAKING PROCESS 

While intelligence analysis can be a complex decision 
making process involving human judgement, intelligence 
analysis has not been "intensively investigated by 
psychologists for more than thirty years" in the manner human 
judgement and decision making have been (Tolcott et al 1989). 
A preliminary literature review reveals somewhat sporadic 
research in the area of intelligence analysis, with the 
majority of the material focused on developing decision 
support aids for the intelligence analyst. 

In 1974 Patten attempted to initiate a method of 
organizing data and information received by an intelligence 
system to help analysts. Patten's concern was the following: 





[that] intelligence analyst[s] working on specific 
problems must be able to selectively obtain information 
relevant to a specific problem without becoming bogged 
down in irrelevant information." (Patten, 1974) 

Patten's study focused on the information and data relevant to 

the analysis process, but not the analysis process itself. 

Sticha, Patterson, and Weiss researched -and determined 

Approximate Reasoning Methods for Decision Aids in 1982, which 

led to the development of a prototype decision aid for Air 

Force target nomination. The reason for studying approximate 

reasoning methods was to hopefully "...achieve a system that 

could facilitate an intelligence analyst's efforts" (Sticha et 

al, 1982), Hence, the actual intelligence analysis process 

became the focus of the research. 

An Intelligence Aid for Estimating Enemy Courses of Action 
(AI/ENCOA) was developed in 1985 in an attempt to provide 
cognitive support to intelligence analysts by altering 
decision rule bases (Lehner et al 1985). AI/ENCOA software 
was developed as a prototype decision aid to assist Army 
tactical intelligence analysts assigned with choosing feasible 
enemy courses of action. This prototype utilized Multi- 
Attribute Utility (MAU) models and enabled the analyst to 
assign weights to each alternative. 

Another decision support software package created for Army 
intelligence analysts was DINAA: The Dynamic Intelligence 
Assessment Aid , by Probus and Donnell, in 1986. DINAA allowed 
the analyst to determine probabilities for each avenue of 
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approach as determined by AI/ENCOA. DINAA utilized Bayes' 
theorem to adjust and assess probabilities. The Prototype 
Analyst Workstation (PAWS) developed in 1990 was similar to 
AI/ENCOA in that it allowed the analyst to define alternative 
courses of action open to the enemy, with the aid of MAU (D. t 

Thompson et al, 1990). 

Cohen, Laskey, and Tolcott developed a prototype decision 
aid in 1987 that could be personalized for each user. 

Submarine commanders were used as the subjects, and the idea 
that 

individual decision makers differ both among 
themselves and from task to task in the 
decision strategy they prefer (Cohen et al, 

1987) 

was examined. This "personalized and prescriptive" decision 
aid allowed the submarine commander to examine different 
decision methods, weights, and trade-offs for each decision. 

Tolcott, Marvin and Lehner conducted research in 1989 in 
"...an attempt to further the investigation of cognitive 
behavior underlying intelligence analysis" (Tolcott et al, 

1989). This study revealed that an intelligence analyst will 
tend to "[remain} with [his/her] original estimate" of a 
scenario, regardless of update information presented (Tolcott 
et al, 1989). It appears as though military intelligence ^ 

analysts will "...pay more attention to supportive than to 

% 

contradictory evidence," as it pertains to their original 
decision/estimate (Tolcott et al, 1989). This research 
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elicited another research question: if update information 
could be defined as feedback, and if feedback could be further 
delineated into different types of feedback, would an 
analyst's perception of update information, or feedback, be 
affected by the specific type of feedback provided? This 
question is addressed in section II.C, Feedback and its Role 
in the Decision Making Process. 

1. Decision Making and Intelligence Analysis 

The process of intelligence analysis is comparable to 
the standard decision making process but even more pronounced 
due to the high levels of stress, dynamic environments, and 
the crucial nature of the decisions arrived at. After 
evaluating large amounts of incoming data (Adelman et al, 
1984), the analyst must "provide information on which 
commanders' decisions are made," validating the analysts' 
position as a critical one in the chain of intelligence and 
operational events (Lewis and Copeland, 1982). It is clearly 
recognized that the analyst routinely faces an intricate and 
complex task that may be gargantuan in its proportions (Luckie 
et al 1968). Essentially, the analyst is tasked with 
repeatedly "quantifying assessed probabilities" (Probus and 
Donnell, 1986). Very often, the analyst is relying solely on 
intuitive feelings, with little or no substantiated feedback 
on his/her decision making process. 


2. Intelligence Analysis as a Process 

Naval Intelligence analysis utilizes the decision 

making process. Intelligence analysis can therefore be 

studied with the lens model and conceptual judgement model, 

and may yield improved performance results with the provision 

of feedback to the analyst. Effective intelligence analysis 

can be defined as an internal, concept-driven process vice an 

external, data-driven process (Katter et al, 1979), and thus, 

as an internal process, has the needs of conceptual judgement, 

including feedback (Hogarth, 1987). 

The intelligence analyst is continually faced with making 
probability assessments concerning both categorical events 
and events which lie along a continuum (Barclay and 
Randall, 1975). 

Therefore the decision making process used by the analyst 
could be ideally studied with the aid of Brunswik's 
interpretation of the lens model. 


C. FEEDBACK AND ITS ROLE IN THE DECISION MAKING PROCESS 

Technically, feedback is the process by which an 
environment returns to individuals a portion of the 
information in their response output necessary to compare 
their process strategy with a representation of an ideal 
strategy (Balzer et al, 1989) 

Feedback, as pictured simplistically by Hogarth in his 

conceptual judgement model, is essential to the decision 

making process. Feedback is relevant to the decision making 

process and it is widely accepted that 

Better performing individuals would be more likely to 
access and use feedback information (Jacoby et al, 1984) 


15 





It is fair to conclude from this that feedback could play an 
important role in improving intelligence analysis. There are 
many varieties of feedback, such as outcome and cognitive. 
What types of feedback will best suit the process of 
intelligence analysis? The answer becomes clear when the two 
most basic types of feedback, outcome and cognitive, are 
examined. 

1. Outcome Feedback 

Outcome feedback is the most basic type of feedback, 
and can be described as the results of a decision process that 
has already occurred. The results are the outcome(s) of 
decisions previously made; hence outcome feedback of a current 
decision does not facilitate changing or altering the current 
decision. Outcome feedback provides information on previous 
decisions only; current decisions are unaffected by outcome 
feedback from their results. Outcome feedback has been 
challenged for its validity as a type of feedback, as it can 
only enhance future decisions vice the current decision making 
process (Balzer et al 1989). Several research studies 
demonstrate that outcome feedback may have little to no 
utility in an uncertain learning environment (Balzer et al 
1989). Hence, outcome feedback may not be optimal in the 
intelligence analysis environment, as intelligence analysis 
frequently occurs in an uncertain environment. 
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2. Cognitive Feedback 

Cognitive feedback provides the decision maker with 
information about the decision currently being made. 
Cognitive feedback allows the decision maker to know what 
their consistency rating is, what weights they have assigned 
to cues, and what weights the environment has assigned to the 
cues. Cognitive feedback consists of three elements; Task 
Information (TI), Cognitive Information (Cl), and Functional 
Validity Information (FVI) (Balzer et al 1989). Cognitive 
Feedback provides what outcome feedback cannot: information 
on the decision making process as it occurs. "Whereas outcome 
feedback is information that describes the accuracy or 
correctness of the response, cognitive feedback represents 
information regarding the how and why that underlies this 
accuracy" (Jacoby et al, 1984). 

Naval Intelligence analysis occurs in a very dynamic 
environment. Since it has been suggested that outcome 
feedback is dysfunctional in a dynamic environment (Jacoby et 
al 1984), and that the type of feedback believed to be most 
beneficial, most often, is indeed cognitive (Balzer et al 
1989), perhaps the feedback with the highest utility for Naval 
intelligence and cryptological analysts could be cognitive 
feedback. Cognitive feedback appears to hold promise for 
improving the quality of intelligence analysis. This could be 
of the utmost importance since many decisions made by analysts 
can affect the safety and lives of others. 




D. HYPOTHESES 


Because of the importance of decision making 
in intelligence analysis, it is imperative 
that more research be devoted to the types of 
decisions that analysts have to make... what 
are optimum feedback mechanisms? (Thompson et 
al, 1984) 

This research examined the following hypotheses in an effort 

to answer the previous question: 

Cognitive feedback is more effective than outcome feedback 
in intelligence analysis tasks. 

Intelligence analysts provided with cognitive feedback 
during their decision making process will exhibit improved 
performance and will make optimal vice satisficing 
decisions. 

The research question was examined through an experiment, 
using Brunswik's interpretation of the lens model and 
mathematical analysis. 

It was imperative to first determine whether or not the 
subjects were affected by cognitive feedback in any way. 
Further analysis determined exactly how the analysts were 
affected: positively, in that they were making more accurate 
and consistent decisions; or negatively, in that they were 
making decisions which were increasingly inaccurate and 
inconsistent with previous decisions. The effect of cognitive 
feedback on Naval Intelligence and Cryptological analysts was 
the focus of this study. 
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Ill. 


METHOD 


A. INTRODUCTION 

Naval Intelligence analysts make complex decisions on a 
daily basis, whether their analysis is as an individual 
analyst for an operational command such as a squadron or ship, 
or they are part of a large watchstanding team that 
collectively analyzes data for the Department of Defense 
(DOD). Naval Intelligence analysis routinely involves complex 
decision making and provides a viable arena for examining the 
effects of cognitive feedback on complex decision making 
processes. 

The research question was investigated in an experimental 
setting designed to most closely represent the type of 
decisions made by analysts in the Fleet. A realistic and 
fairly complex task of moderate to difficult complexity was 
created in an effort to employ the broadest range of the 
analyst's decision making capabilities. 

B. Experimental Design 

An experiment involving comparison of results between two 
sets of subjects was devised. One set of subjects (12 
subjects) was given outcome feedback only; another set (12 
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subjects) was given cognitive and outcome feedback. The 
presentation of feedback was designed to provide an optimal 
amount of information to the subject with minimal complexity 
in the presentation format. 

The subjects for each part of the experiment were randomly 
assigned to receive outcome or outcome and cognitive feedback. 
Subjects receiving cognitive feedback had a longer instruction 
set to read and understand before they were able to begin the 
experiment. Outcome feedback was presented as accuracy, 
whereas cognitive feedback was presented as Decision Support 
Information. Both types of feedback, and how they were 
designed and presented to the subject, are described in the 
following sections. 

C. PARTICIPANTS 

Subjects chosen for this experiment were all on active 
duty in the United States Navy. They are currently serving, 
or have served, in positions requiring intelligence or 
cryptologic analysis. Both Enlisted Personnel and Commissioned 
Officers were used in the experiment. All subjects have 
received training by U.S. Navy schools, and have had 
experience at making intelligence decisions. The actual 
positions and billets held by the experiment's subjects 
varied, from operational intelligence billets afloat, to 
standard administrative-type billets ashore. 
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TABLE 2. EXPERIMENT SUBJECT DEMOGRAPHICS 


Demographic 

Mean 

Standard Deviation 

Age (years) 

29.0 

7.0 

Rank (E-1 through 
0-9) 

9.0 (E-9) 

3.7 

Analysis Experience 
(years) 

5.5 

4.6 

How long since 
conducted analysis 
in the Fleet 
(years) 

1.5 

0.9 


The subject demographics were calculated with the 
assistance of Minitab (Schaefer and Anderson, 1989). 


D. TASK 

The task of "quantifying assessed probabilities" (Probus 
and Donnell, 1986) to various cues or pieces of information is 
central to the intel1igence analyst's decision making process. 
This is academically and professionally acknowledged as a 
difficult task, and has been examined and incorporated into 
the development of different systems to aid intelligence 
analysis, such as The Dynamic Intelligence Assessment Aid 
(DINAA) designed by Probus and Donnell in 1986; and described 
in the User's Manual for the Prototype Analyst Workstation 
(PAWS) by Thompson, et al , in 1990. Both DINAA and PAWS 
represent attempts to first capture, and then aid, the process 
of intelligence analysis. This highlights the first challenge 
faced when attempting to prove the hypothesis of this study: 
the experiment must include a task similar in complexity to 



















the task performed by a Naval Intelligence analyst on a daily 
basis. The validity of the experiment hinged upon the data 
collected from the analysts, and the data could only be as 
realistic and meaningful as the task itself. 

1. Selection of Task 

The task of intelligence analysis requires the analyst 
to provide a decision based upon whatever information is 
available. To represent this task and ensure the robustness 
of the experiment, a task similar to that performed by 
analysts in the Fleet was devised. The task requires the 
analyst, or subject, to provide his/her best estimate of a 
suspect ship's position, based upon given information. The 
given information was dependent upon whether or not the 
subject had access to solely outcome feedback, or cognitive 
and outcome feedback. 

2. Description of Task 

The actual task presented to the analysts involved an 
anti-drug smuggling situation. Anti-drug smuggling tasking 
was chosen due to its relevancy in military operations and 
intelligence. The task was first presented to the analyst 
from a political standpoint of the situation in the Backgroimd 
Information. The Background Information provided current 
domestic and international political and social information as 
it pertained to the influx of illegal drugs into the United 
States. The purpose of the Background Information was to 
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create a basis of familiarity for the analyst, and to give the 
task a realistic perspective. 

The task was reduced to a single situation, similar to 
the analysts daily tasks in the Fleet, in the Situation Report 
and Scenario, both which narrowed the scope of the illegal 4" 

drug trafficking trade to a single suspect vessel. The 
analyst was asked to provide his/her best estimate of the 
location of the suspect vessel based upon the information 
provided. 

Basic information provided on the suspect vessel 
included that it had completed three previous runs from 
Colombia to the U.S. over the past six weeks, tracking east of 
Cuba the first time, and west of Cuba the last two times. The 
analyst was told the vessel was estimated to carry up to three 
tons of cocaine, and that it was preparing to depart Colombia 
once again for the U.S. mainland. This basic information on 
the suspect vessel provided the analyst with a brief history 
of the vessel, the general geographic location of the 
scenario, and an idea of the vessel's drug carrying 
capabi1ity. 

3. Task Variables/Cues 

A realistic task necessitates variables and random 
weighting of the variables. This task incorporated four 
variables: weather, ship type/speed, presence of the Cuban 

Navy, and the last contact information on the suspect vessel. 
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All four variables were present throughout the task; thus the 
analyst had information regarding each of the four variables 
each time he/she was asked to perform the task of providing an 
estimated suspect vessel position. Each variable appeared to 
be weighted differently in each scenario presented to the 
analyst, and the combination of the differently weighted 
variables varied as well. This was to provide as realistic a 
task as possible, and to optimally test the analysts decision 
making capabilities. However, the actual position of the 
ship, as calculated by the system, weighted the ship 
type/speed, weather, and last contact variables equally and 
included a random error. The proximity of the Cuban Navy 
variable was negligible in the calculation of the ship's 
actual position. The values used for each of the variables as 
presented to the analysts are represented in Table 3. 
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TABLE 3. EXPERIMENTAL CUE CHARACTERISTICS 


Variable 

Values 

Last 

Contact 

1-12 hours 

Ship 

Type/Speed 

Sailboat (6-8 knots) , 

Yawl (10-12 knots) 

Cargo Ship (19-20 knots) 

Speedboat (40+ knots) * 

Sea State 

1 = Calm, winds up to 10 knots 

2 = Breezy, winds up to 20 knots 

3 = Heavy swells, winds up to 45 

knots 

4 = Hurricane force, gale winds 

up to 80 knots 

Cuban Navy 
in Area 

20 nautical mile radius 

15 nautical mile radius 

10 nautical mile radius 

3 nautical mile radius 


E. PRESENTATION 

1. Task Description/Information 

All subjects, regardless of the type of feedback 
provided to them, received the same basic information to 
enhance the naturalness of the experiment. Background 
Information and Situation Report provided geopolitical 
information on the areas of concern for DOD and the United 
States Government. In this experiment, Background Information 
stated that the focus of the President is on the anti-drug 
efforts of the U.S. This set the stage for the scenario in 
which the subject was asked to provide a position for the 
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suspect vessel. Scenario provided information as to what 
billet the subject was to place him/herself in, and for what 
purpose they were to provide an estimated position of the 
suspect vessel. 

keeping with the experimental rule of "ten 
scenarios per cue" (Balzer and Doherty, 1991), the analyst was 
asked to provide an estimated position of the suspect vessel 
forty separate times during the experiment. Each scenario was 
independent of any other, and incorporated the four variables, 
or cues, provided to the subject. The four variables were 
last contact, ship type/speed, weather, and Cuban Naval 
exercise proximity to the location of the suspect vessel. The 
actual values of each variable (hours, knots, sea state, 
proximity of Cuban Navy in nautical miles) were varied in each 
scenario. Thus each scenario used the same four variables, 
but in a different combination of values for each instance. 

Variables were presented to the analysts in the form 
of message sets, which are very familiar to intelligence 
personnel. An example of a message set is in the Appendix. 

Since the entire experimental scenario and background 
information was unclassified, so were the message sets, the 
origin of each message, and its destination. 

The four variables; last contact, ship type/speed, 
weather, and Cuban Naval activity proximity to the suspect 
vessel, were all presented in the message set. The analyst 
was presented with forty message sets in total, each formatted 
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identical to the sample message set in the Appendix, but with 
the values of the cues, or variables, randomly varied. 

2. Geographic Representation 

The task was designed to best represent the tasks 
performed by analysts on a daily basis in the Fleet. The task 
asked the analyst for his/her best estimate of the suspect 
vessel's position, based upon the information presented in the 
respective message set. In a real situation, the analyst 
would be asked to provide an estimated position in degrees 
latitude and longitude. To provide a more straightforward 
representation of the task and also to place necessary limits 
on the pertinent geographic area of the task, a grid square 
was designed to cover the geographic area of the experiment. 

The analyst was asked to locate the suspect vessel in 
an area in the Caribbean, north of Colombia and Venezuela. 
The geographic area of the task is depicted in the Appendix. 
The grid square was bounded by the following four coordinates: 

12N 72W; 18N 72W; 12N 78W; 18N 78W. The grid square was 
divided into thirty-six smaller squares, and each square was 
approximately sixty by sixty nautical miles. Each square was 
assigned a two-digit, unique number. This allowed for greater 
ease in providing a position, and thus greater ease in 
calculations involving the positions of the suspect vessel. 
The grid square did not, however, detract from the continuous 
nature of the latitudinal and longitudinal scales of 




measurement. Continuous values for the position(s) of the 
suspect vessel were preserved with the grid square as each 
position remained unique and numerically higher or lower than 
any other grid square. The grid square numbering maintained 
the continuity of the number line. 

What was most realistic in the experimental setting 
was that the message sets presented multiple cues to the 
subjects in a format familiar to the subjects. This enhanced 
the viability of the results captured by the experiment, as it 
allowed the analysts to complete a task as familiar as 
possible to them. Experiment debriefing revealed that nearly 
every subject believed the experiment and task to be highly 
realistic and believable. 

F. SETTING 

Subjects received experiment Background Information, 
Scenario, Situation Report, Message Sets, Computer Instruction 
Set, and Decision Support Information manually, in a loose- 
leaf binder. The experimenter presented the subject with the 
binder after briefly explaining the geographical area of the 
experiment with the aid of a navigational chart. Background 
Information, Scenario, Situation Report, and Message Sets were 
described in sections III.D and III.E. Computer Instruction 
Set and Decision Support Information will be described in this 
section, and section III.G, respectively. 





1. Geographic Area 

A navigational chart identical to those used in the 
Fleet by analysts conducting actual tasks was provided, with 
the task area delineated in grid format. Thirty-six squares, 
approximately 60 nautical miles X 60 nautical miles each, were 
numbered to represent a continuous scale similar to latitude 
and longitude. Each square was unique. The grid-square 
numbering system allowed for greater ease in statistical 
analysis without detracting from the actual continuous 
latitude/longitude system employed in the Fleet. 

The geographic area of the experiment, defined by the 
grid square, was covered with acetate to allow marking of the 
area by the analyst with a water soluble marker. Rulers or 
navigational plotters/dividers were provided to the subject 
for distance calculation. 

Each subject was provided with a piece of scrap paper 
and a pencil or pen for any time-distance calculations 
required. Calculators were not permitted as a general rule. 

2. System 

The experiment was conducted on IBM compatible 286/386 
computers, in private office or computer laboratory areas. 
Each subject was given a one page Computer Instruction Set 
which explained the screen on which the vessel's estimated 
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position, in the form of a grid square number, would be 
entered. Instructions on how to receive outcome or cognitive 
feedback were included in the appropriate instruction set. 

Additional computer instructions were presented to the 
subject on the screen of the computer they were using, 
reminding the analyst to consider each message set 
individually, and to enter each two-digit estimated position 
of the suspect vessel into the computer after they had 
analyzed each message set. The last instruction on the screen 
directed the subject to "strike any key when ready," upon 
which time the four-column screen for position entry was 
displayed on the monitor. 

3. Data Capture/Entry 

The system captured the subject's estimated positions 
when the subject entered their estimated position for each 
message set. The subject was presented with a monochrome 
screen with four columns of ten entry places each. To the 
greatest extent possible, a graphical display was used that 
had been found to be effective for approximate comparison of 
quantitative information (Brehmer, 1984). The entry places 
for the subject's estimated positions were numbered 1-40, and 
further computer instruction information was presented on the 
bottom line of the screen. The columns for data entry 
covered the bottom-half of the screen. The top-half of the 
screen remained blank for subjects receiving outcome feedback 






only. When the subjects receiving solely outcome feedback 
pressed "END," they were presented with the actual position of 
the the suspect vessel (accuracy, or r^,) in the right hand 
columns next to their estimated vessel position. 

The subjects receiving cognitive feedback were presented 
with a menu after they pressed "END." The menu was displayed 
in the top half of the screen. After choosing a cognitive or 
outcome feedback option, the subject participating in the 
cognitive feedback experiment was provided with decision rule 
scales in the top half of the screen, or consistency and/or 
accuracy information in the columns preceding or following the 
estimated position entry column. 

An experimenter was present for the entire length of the 
experiment, to provide guidance to the subjects and ensure 
proper task completion by the subjects. The subjects were 
given no time limit for completing the forty tasks involved in 
the experiment. Subjects took anywhere from 50 minutes to 135 
minutes to complete the experiment. 

G. DESIGN OF FEEDBACK 

1. Outcome Feedback 

In the experiment version that incorporated outcome 
feedback only, the computer instruction set indicated that the 
analyst could access outcome feedback, in the form of the 
suspect ship's actual position, by pressing the "END" key on 
the keyboard. Once the subject had accessed outcome feedback. 





he/she was unable to change any positions he/she had already 
entered. 

In the experiment incorporating both outcome and 
t cognitive feedback, the analyst was able to access their 

accuracy, or outcome feedback, by selecting the "accuracy" 
menu choice on the feedback menu. 

Outcome feedback was presented in similar fashion to 
the subjects for both sets of the experiment. When the 
subject accessed "accuracy," the outcome feedback was 
displayed, in the form of a two-digit suspect vessel position, 
as estimated by the system. The system's estimate of the 
suspect vessel's position appeared in the right-hand column of 
the data-entry screen, next to the suspect vessel's position 
estimated by the subject. The outcome feedback, in the format 
of "accuracy," is displayed in Figure 3. 

Outcome feedback, or accuracy, was calculated with the 
use of consistency and predictability information. 

2. Cognitive Feedback 

All subjects conducting the outcome and cognitive 
version of the experiment received an additional instruction 
set. Decision Support Information, which detailed the five 
choices of cognitive, or "decision" feedback available to the 
analyst. 


32 



(Sengupta, 1990) 


To access both cognitive and outcome feedback, the 
analyst pressed the "END" key, which then provided them with 
a menu of information feedback options. Choices numbered 1 
through 5 on the menu provided cognitive feedback information. 
Choice number 6 provided outcome feedback, in the form of 
accuracy, as described in the previous subsection. 

The experimental model required that the analyst had 
provided positions for at least 15 message sets before any of 
the cognitive feedback options could be accessed. Cognitive 
feedback information was presented to the subject in the 
following formats: 

1. Decision Rule Information: This provided specific weights 
assigned to the variables by the analyst for the previous 
scenarios which the analyst had provided an estimated 
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position for. This function would reveal to the subject 
which of the variables they were placing the most and 
least values on to make their decision. The calculation 
of the weights assigned to the variables was accomplished 
by the following method (as in Sengupta, 1990): 

(a) Weights assigned to the variables were calculated by 
a multiple regression of variable values and the analyst's 
estimation of the weights. 

(b) The weights were then transformed into a percentage 
representation of the total value of the weights. 

(c) The weights, transformed into percentages, were then 
displayed on a horizontal bar graph as shown in Figure 4. 



2. Consistency Information: This function provided the 

subject with the positions they would have chosen for the 
suspect vessel if they were indeed applying their decision 
rule consistently. As with choice one. Decision Rule 


34 



Information, only the message sets for which the analysts 
had provided an estimated position could be given 
correlated consistency positions. Consistency was 
calculated by a multiple correlation between the variables 
and the analyst's estimation of the weights of the 
variables. Consistency was presented to the analyst on 
the data entry menu, in the left-hand column next to the 
analyst's estimated suspect vessel positions. An example 
of the presentation of consistency is depicted in Figure 
5. 



3. Information on the System's Decision Rule: This function 
provided the subject with the actual weights for each 
variable, or cue, as applied by the system, to achieve the 
correct, or system, answer. This was calculated in a 
manner similar to Decision Support Information, but the 
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multiple regression was between the actual weights of the 
variables, and the assigned values of the variables. 
Information on the system's decision rule was represented 
on a horizontal bar graph. 

4. Information on the System's and the Subject's Decision 

Rule: a combination of choices one and three, this 

function allowed the subject to graphically compare 
his/her applied decision weights with the weights utilized 
by the system. 

5. Information on the Subject's Decision Rule and 

Consistency: a combination of choices one and two, this 

function presented the subject with both the weights 
he/she was utilizing, as well as his/her degree of 
consistency. 

6. Accuracy: this function served as the outcome portion of 
this version's feedback, as it provided the subject with 
the actual vessel position. As in the outcome feedback 
only version of the experiment, once the subject accessed 
accuracy, or outcome feedback, he/she was not able to 
revise any previously entered vessel positions. 
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IV. EXPERIMENTAL RESULTS 


A. TASK RESULTS 

The statistical analysis of the experiment supports the 
hypotheses by revealing that the analysts presented with 
cognitive feedback and outcome feedback performed 
significantly better than the analysts who received outcome 
feedback only. Hence, cognitive feedback impacted the 
intelligence analysis process pertinent to the experimental 
task in a positive manner. 

An immediate and clear measure of the performance of the 
analysts is represented in Table 4. 


TABLE 4. MEANS AND (STANDARD DEVIATIONS) OF PERFORMANCE 


TYPE OF 

FEEDBACK 


Rs 

G 

Cognitive 

Feedback 

.841 (.106) 

.914 (.068) 

.967 (.041) 

Outcome 

Feedback 

.658 (.192) 

.811 (.077) 

.717 (.180) 


The measures of performance are the achievement index, or 
accuracy, r., the consistency index, Rj, and the matching 
index, G. In all three measures, the analysts receiving 
cognitive feedback performed markedly better than the analysts 
receiving outcome feedback only. 
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To more closely examine these results, an analysis of 
variance (ANOVA) amongst the performance factors was 
conducted. This was calculated using the General Linear 
Models procedure in SAS (SAS, 1987). The results are shown in 
Table 5. 


TABLE 5. ANOVA OF MODEL AND (ERROR) 


Dependent 

Variables 

Degrees 

of 

Freedom 

Sum of 
Squares 

Mean 

Square 

F Value 

Pr > F 


1 

(22) 

4.720 
(1.904) 

0.079 

(0.019) 

4.05 

0.0001 


1 

(22) 

0.624 

(0.398) 

0.010 

(0.004) 

2.56 

0.0001 

G 

1 

(22) 

2.606 

(1.968) 

0.043 

(0.020) 

2.16 

0.0003 


The results indicate that subjects receiving cognitive 
feedback had a significantly higher achievement score than 
those receiving outcome feedback (F(l,22) = 4.05; p < 0.0001). 
Subjects receiving cognitive feedback had a significantly 
higher consistency score than those receiving outcome feedback 
only (F(l,22) = 2.56; p < 0.0001). Subjects receiving 
cognitive feedback also had a significantly higher matching 
index than those receiving outcome feedback only (F(l,22) = 
2.16; p < 0.0003). We thus conclude that subjects receiving 
cognitive feedback performed better than those receiving 
outcome feedback only. The null hypothesis is, therefore, 
rejected. The statistical results of the ANOVA of the 
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performance indicators also support the hypothesis that 
cognitive feedback positively affects the performance of 
intelligence analysts. 

B. DEBRIEF RESULTS 

The debriefing questionnaires and task block surveys 
revealed information on the subjects perceptions of the 
experiment. Overall, the analysts found the task to be 
difficult, and felt that it moderately resembled a task they 
had previously performed in the Fleet. Subjects found the 
task to be clear, and took the experiment very seriously. 

Analysts who received cognitive feedback found it to be 
moderately helpful, and all subjects felt that the outcome 
feedback (actual ship's position) was moderately helpful. 





V. CONCLUSIONS 


A. SUMMARY 

The purpose of this study was to evaluate the efficacy of 
cognitive feedback in improving intelligence analysts' 
performance. The statistical results of the experiment 
indicate that analysts who received cognitive feedback did 
indeed perform better than their colleagues who received only 
outcome feedback. 

B. CONTRIBUTION 

Evidence thus indicates that cognitive feedback positively 

impacts performance. The significance of this study is that 

it involved U.S. Naval Intelligence and Cryptologic personnel, 

and indicates that cognitive feedback could be beneficial to 

more types of intelligence analysis. For example, 

intelligence analysts tracking a suspect vessel would be able 

to receive information on the significance of the cues in the 

particular environment in which they are working 

(environmental weights). Analysts could have, at a glance, a 

« 

summary of the emphasis that they have been placing on various 
cues in a particular problem (decision weights). Analysts 
could also access information relating how consistent their 
choices are (consistency). One or all of these added tools to 



the environment of intelligence analysis could.be beneficial 
and lead to improved analysis results. 

C. FUTURE RESEARCH 

Future research could use this study as a point of 
departure to determine which forms of cognitive feedback are 
most beneficial to intelligence analysis. This could lead to 
the development of a Decision Support System prototype for 
U.S. Naval Intelligence activities. 


41 


APPENDIX 


NAVAL INTELLIGENCE OFFICERS; A LENS VIEW OF INTELLIGENCE 
ANALYSIS 

INSTRUCTION SET 

A. Introduction 

1. The purpose of this study is do examine how Naval 
Intelligence Officers and Specialists process complex 
information. 

2. Your participation in this experiment is voluntary and 
the results of your participation will be used for research 
purposes only. Please take the experiment seriously and 
attempt to answer the queries as realistically as possible. 

B. Your Task 

1. Your task involves three basic, steps. They are: 

a. Read the background information for your scenario 
provided in your task booklet. You may need to read it over 
two or three times before you feel thoroughly comfortable with 
it. The scenario will provide information regarding the task 
you are asked to perform. For example, you will be told the 
history of a particular drug-smuggling ship that you will 
later be asked to provided an estimated location for. 

b. Examine, individually, each of the 40 message packages 
provided. Each message set is on a separate piece of paper 
and consists of three "messages" similar to those you use in 
the Fleet. The scenario will take place in the Caribbean. 
Each message set will provide information on the ship's 
type/speed, the last contact (in hours), weather in the 
Caribbean, and Cuban Naval presence in the vicinity of the 
suspect vessel. 

A typical message set is composed of three messages 
whose bodies of information appear as follows: 

"SUSPECT SHIP LOCATED 15 HOURS AGO IN THE VICINITY OF GRID 
SQUARE # 373, TRAVELING AT 3 KTS." 

"WEATHER REPORT FOR 15 JANUARY: CLEAR SKIES, WINDS FROM 

THE WEST AT 10-12 KTS, NO WHITECAPS, MINIMAL WAVES." 





"CUBAN NAVAL UNITS SIGHTED WITHIN 1 NM OF SUSPECT VESSEL." 


c. After you read and analyze each message set, you are 
to provide a grid square position number of where you best 
estimate the target unit to be located. You will use the 
chart provided and the navigation set to determine this. 


REMEMBER--each message set is independent of all others, 
hence you will estimate a new position for each message set. 

You will complete two blocks of 40 queries, for a total of 
80 position estimates. 


d. This is a not a simple task and the answer may not 
seem "easy" to arrive at. Do not get discouraged, and 
remember that you were chosen for this experiment because of 
your expertise at decision-making in a complex environment. 

2. Please read the computer instructions carefully. You will 
receive the scenario background information and message 
packages manually, and enter your estimated position into the 
computer. You may also be able to use the information on the 
computer to receive feedback on your decisions, and this will 
be specified in the computer instructions. 

3. After you complete the entire task (both blocks), please 
fill out the debrief questionnaire and the demographic 
information sheet provided. Please be honest and as clear as 
possible in your responses. Turn your entire packet into the 
experimenter. 


Cognitive Feedback 


COMPUTER INSTRUCTION SET 

* You will be presented with four columns on your screen. 
There are ten rows in each column, one for each message set, 
for a total of forty message sets per block. 

* The message sets will be presented manually. Each set is on 
a separate piece of paper. The message sets incorporate 
information that will enable you to estimate the target ship's 
location. Specifically, you will be given information on the 
ship type/speed, last contact (in hours), weather, and Cuban 
Naval presence in the area. 

* You can enter your grid square number choice for each 
message set next to the appropriate message set number. You 
need not analyze the message sets in any particular order. 
You can maneuver about the screen with the cursor. The "up" 
and "down" arrows allow you to select the message sets you 
choose to analyze, and also allow you to change answers if you 
choose to. 

* At anytime during the experiment you may access decision 
support information on the accuracy your decision making 
process. Simply press "END" and a decision support 
information menu will be presented to you at the top portion 
of your screen. You may select up to six different types of 
feedback, but once you chose feedback that displays the 
accuracy of your decisions, you may not change any decisions 
already completed. You will receive further information on 
the decision support information and how to use it before you 
begin the experiment. 

* All decision rule information requires that you have 
entered at least 10 positions before you can access this 
information. 

* After you have completed Block One, complete the Block One 
Questionnaire and press "HOME" to move to Block Two. 

* After Block Two, complete the Block Two questionnaire, the 
task debrief survey, the demographic information sheet, and 
inform the experimenter that you are finished. 

* The line at the bottom of the screen will provide you with 
information on how to maneuver about the screen and how to 
receive additional information. 

* IF AT ANYTIME DURING THE EXPERIMENT YOU ARE NOT SURE OF HOW 
TO USE THE PROGRAM, ASK THE EXPERIMENTER. 






Outcome Feedback 

COMPUTER INSTRUCTION SET 

* You will be presented with four columns on your screen. 
There are ten rows in each column, one for each message set, 
for a total of forty message sets per block. 

* The message sets will be presented manually. Each set is on 
a separate piece of paper. The message sets incorporate 
information that will enable you to estimate the target ship's 
location. Specifically, you will be given information on the 
ship type/speed, last contact (in hours), weather, and Cuban 
Naval presence in the area. 

* You can enter your grid square number choice for each 
message set next to the appropriate message set number. You 
need not analyze the message sets in any particular order. 
You can maneuver about the screen with the cursor. The "up" 
and "down" arrows allow you to select the message sets you 
choose to analyze, and also allow you to change answers if you 
choose to. 

* At anytime during the experiment you may find out what the 
correct answers were for message sets you have already 
completed. By pressing "END" you will be provided with the 
"correct" ship positions. Once you access this information, 
you may not change any answers you have already entered. 

* After you have completed Block One, complete the Block One 
questionnaire and press HOME" to move to Block Two. 

* After Block Two, complete the Block Two questionnaire, the 
task debrief survey, the demographic information sheet, and 
inform the experimenter that you are finished. 

* The line at the bottom of the screen will provide you with 
information on how to maneuver about the screen and how to 
receive additional information. 

* IF AT ANYTIME DURING THE EXPERIMENT YOU ARE NOT SURE OF HOW 
TO USE THE PROGRAM, ASK THE EXPERIMENTER. 






BLOCK ONE 


Background Information 

The United States has been fighting the drug war for over 
three years, yet the effectiveness of the war is questionable. 
The Presidential election is only one year away, and the 
current administration wishes to ensure continual funding for 
the anti-drug effort. 

World politics have taken a backseat to domestic issues. 
The current administration realizes that domestic policy has 
not been its strong suit in the press, as presented to the 
American people. One issue that is repeatedly 
spotlighted by the media is that of the "crack babies" being 
born to cocaine addicted mothers, at an alarming rate. Crack 
babies are not just born in the inner-city ghetto atmosphere, 
but to mothers of all social strata in the U.S. Recent news 
commentaries have revealed that the importation of cocaine and 
crack into the U.S. is at an all time high. Congress is 
beginning to ask what is going on with the drug war. 

Senior military officials know that their anti-drug 
campaign has been relatively successful and fruitful. In 
light of the small number of U.S. armed forces dedicated to 
the anti-drug campaign, the total amount of captured 
contra-band is phenomenal. But in comparison to the total 
amount of cocaine and crack smuggled into the U.S. annually, 
it is barely the tip of the iceberg. 

The President saw a need to step up the intensity of the 
drug war. He elicited advice from his top military personnel 
regarding just how much military hardware and personnel would 
be needed to accomplish this. Then, the President went to 
Congress with the Secretaries of Defense and Transportation 
and presented a proposal to attempt to finally put a noose on 
the cocaine flow from Colombia, Bolivia, and Peru. The 
country of chief concern is Colombia, since drugs harvested in 
Peru and Bolivia are transported through Columbia and then 
flown or shipped to the U.S. 

Situation Report 10 November 1991 

In a sweeping majority* last week. Congress voted to 
increase the budget for the drug war ten-fold. This increase 
was voted in under the conception of the American people that 
the war could now be tackled in a short period of time and put 
an end to the drug trade in the Caribbean. 

Scenario 
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You are on watch in the Joint Team Against Drugs Command 
in Florida. Pressure has been increasing for your watch team 
to get the best intelligence possible out to the Fleet surface 
and air units who are responsible for patrolling marked areas 
in the Caribbean. 

You have received intelligence indicating that a vessel 
capable of carrying three tons of cocaine is preparing to 
depart Colombia for the U.S. mainland. A history of this 
vessel reveals that it has completed three runs from Colombia 
to the U.S. in the last six weeks. Its tracks were around the 
eastern edge of Cuba the first time, then around the western 
edge the last two times. Track navigation took 4 days each 
time. 

You are directed to provide the most likely area the 
vessel is at the current time. Taking into consideration the 
information provided in the most recent message traffic, in 
what area would you send the aircrews and ships? It is 
currently 11 Nov and you need to estimate the vessel's 
position. Please provide a grid square number where you best 
estimate the suspect ship to be. 


Decision Support Information 


* What is Decision Support Information? 

Decision Support Information is diagnostic information 
provided by the system on your decision processes and that of 
the system. You can access the decision support when making 
your decisions on the suspect vessel's positions. (In order 
to compute the information, however, the system needs at least 
10 scores from you). By accessing such information, decision 
makers can derive better insight into their decisions 
processes. This enables them to revise and improve their 
decisions (or positions, in this case) through a "what-if" 
mode of analysis. 


* How do I use Decision Support Information in Making 
Decisions? 

Typically, you the decision maker, would use the decision 
support information as follows: 

1. Make some tentative decisions (i.e., ship's 
positions). 

2. Ask the system for decision support information. 

3. Refine your positions accordingly. 

You may wish to do this 1-3 sequence within a block, as 
many times (and with as many scores) as you wish. 

This sequence allows you to delay asking for your accuracy 
and enables you to continue to refine your positions. 


* Types of Decision Support Information 

(For illustrations of Decision Support Information, see 
III.G.2) 

The system will provide you with the following 5 types of 
decision support information: 

1. INFORMATION ON YOUR DECISION RULE: 

Decision makers are sometimes unable to specify 
precisely, a particular decision rule (in this case, weights 
assigned to specific information regarding the suspect 
vessel). The system will track the weights you are using (in 





formulating your positions) and will display them through a 
stacked-bar chart. 

How do I use it? 

1. Make sure the weights displayed are actually the 
ones you want applied. 

2. If not, revise your scores, and see how the 
weights change. 

3. Iterate between 1-2 until the system shows 
weights you actually want applied. 


2. INFORMATION ON YOUR CONSISTENCY: 

Sometimes, after decision makers have specified their 
decision rules, they are unable to apply them consistently. 
The system will calculate the scores you would have given had 
you been completely consistent with your decision rule. 

How do I use it? 

1. Check your scores against the consistency scores. 

2. Revise your scores if you need or wish to. 

3. Iterate between 1-2 till your scores match with or 
are close to the consistency scores. 


3. INFORMATION ON THE SYSTEM'S DECISION RULE: 

Instead of trying to figure out from several examples what 
rule the system is using, it is more effective if it is 
displayed in a bar graph format. 

How do I use it? 

1. Use the information to get an idea of what decision 
rule the system has been following. 

4. INFORMATION ON THE SYSTEM'S DECISION RULE AND YOURS 

This is actually a combination of decision support 
information 1 and 3. It enables you to compare your decision 
rule with that of the system and thereby emulate the system 
better. 


How do I use it? 


1. Check weights you have given versus weights given by 
the system. 

2. Revise your positions if you need to. 

3. Iterate between 1-2 until your weights match with or 
are close to the system's weights. 


5. INFORMATION ON YOUR DECISION RULE AND CONSISTENCY 

This is actually a combination of decision support 
information 1 and 2. The idea here is to let you revise your 
weights without losing your consistency at the same time. 

How do I use it? 

1. Make sure the weights displayed are actually the ones 
you want applied. 

2. If not, revise your positions, and see how the weights 
change. 

3. Check your scores against the consistency scores. 

4. Revise your scores if you need or wish to. 


> 

***YOU ARE NOW READY TO PROCEED WITH THE TASK*** 
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MESSAGE SET ONE 


FM: USS SPIKE 

TO: JTAD 

SUBJ: VESSEL SIGHTING 

SUSPECT VESSEL SIGHTED BY FANTAIL LOOKOUT APPROX. 7 HRS. AGO, 
AT 

GRID SQUARE #68, MOVING 18 KTS. 


FM: NOAA 

TO: JTAD 

USCOMSOLANT 

SUBJ: WX REPORT FOR 11 NOV 1991 

PARTLY CLOUDY SKIES WITH WINDS FROM THE NNW AT UP TO 45 KTS, 
CAUSING HEAVY SWELLS AND WHITECAPS. 


FM: USS SCUTTLEBUTT 

TO: JTAD 

USCOMSOLANT 

SUBJ: CUBAN NAVAL EXERCISE ACTIVITY 

CUBAN NAVAL EXERCISE UNITS SIGHTED WITHIN 20 NM OF SUSPECT 
VESSEL. 





QUESTIONS TO BE ANSWERED AFTER COMPLETING BLOCK ONE 


cognitive feedback 


1. Describe (in words or equations) what decision rule you 
followed in making your own estimates: 


2. Distribute 100 points among the three variable you used 
for reaching your overall estimate-in accordance with the 
importance you assigned them (total will add up to 100). 

Ship's speed _ 

Ship's last contact _ 

Weather _ 

Cuban Naval Presence__ 

100 Total 

3. What do you think the actual weights the system used for 
each of the variables was? 

Ship's speed _ 

Ship's last contact _ 

Weather _ 

Cuban Naval Presence_ 

100 Total 

4. In this task, did you request decision support information 
at any time from the system? Y N 

5. If "YES", try to describe how you used decision support 
information in making your decisions. 


h- 
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QUESTIONS TO BE iANSWERED AFTER COMPLETING BLOCK ONE 

outcome feedback 


1. Describe (in words or equations) what decision rule you 
followed in making your own estimates: 


2. Distribute 100 points among the three variable you used 
for reaching your overall estimate-in accordance with the 
importance you assigned them (total will add up to 100). 

Ship's speed _ 

Ship's last contact _ 

Weather _ 

Cuban Naval Presence_ 

100 Total 

3. What do you think the actual weights the system used for 
each of the variables was? 

Ship's speed _ 

Ship's last contact _ 

Weather _ 

Cuban Naval Presence_ 

100 Total 




4 
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DEMOGRAPHIC SURVEY 

NAME_AGE__SEX 

4 

RANK _DESIGNATOR_ 

COMMISSIONING DATE__COMMISSIONING 

SOURCE_ 

YEARS IN INTELLIGENCE FIELD_ 

PREVIOUS OPINTEL EXPERIENCE (DUTY STATION, DATES) 


ALL OTHER PREVIOUS ASSIGNMENTS 


INTELLIGENCE SCHOOLS ATTENDED (NMITC BASIC, NMITC OPINTEL, 
ETC)___ 


FULL-TIME WORK EXPERIENCE (IN YEARS)_ 

HIGHEST DEGREE EARNED: HIGH SCHOOL_ ASSOCIATE’S_ 

BACHELOR’S_ GRADUATE_ OTHER (SPECIFY)_ 

HOW LONG AGO (IN YEARS) DID YOU COMPLETE YOUR HIGHEST LEVEL OF 
EDUCATION? _ 

HOW FAMILIAR ARE YOU WITH COMPUTERS, GENERALLY? 

1 2 3 4 5 6 7 8 9 

Not at Very 

all familiar familiar 
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HOW MANY HOURS PER WEEK DO YOU USE COMPUTERS?_ 

YOUR GENERAL COMMENTS REGARDING THE EXPERIMENT: 




4 
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TASK DEBRIEF SURVEY 

Cognitive Feedback 


1. How would you rank the difficulty of your task during the 
past 40 scenario snapshots? 


1 2 3 4 5 6 7 

not 

at all difficult 


8 9 

very 

difficult 


2. How would you rank the clarity of your task? 


1 2 3 4 5 6 7 

not 

at all clear 


8 9 

very 
clear 


3. How closely does the task resemble what you do on a 
day-to-day basis in the fleet? 


1 2 3 4 

does not 
resemble at all 


5 6 7 8 9 

resembles 
very closely 


4. Have you performed a similar task in the past? Y N 
a. If "YES" how long ago? (in years/months)_ 


5. To what extent were the concepts of decision support 
information clear to you (as explained by the instructions and 
the experimenter)? 


12 3 

Not clear 
at all 


4 5 6 7 


8 9 

Very 
clear 


6. To what extent was decision support information helpful in 
improving your own decision? 


1 23456789 

Not at all Very 

helpful helpful 

7. To what extent was information about the ship's actual 
position helpful in improving your own decision? 

1 23456789 

Not at all Very 

helpful helpful 


4 


V 
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8. Now that you have completed the task, can you think of any 
other factor (other variables, etc) that may have influenced 
you in making your decision? 




9. How clear were 
1 2 3 

Not clear 
at all 


the instructions regarding the task? 
4 5 6 7 8 9 

Very 

clear 


10. How would you present the task 

differently?_ 


11, How seriously did you take this task, generally? 


1 2 3 4 5 6 7 8 9 


not 

seriously 


very 

seriously 


12. How easy was this system to use? 


1 2 3 4 5 6 7 

Not at all 
easy 


8 9 

Very 
easy 


4 


4 


57 






TASK DEBRIEF SURVEY 


Outcome Feedback 


1. How would you rank the difficulty of your task during 4;he 
past 40 scenario snapshots? 

1 23456789 

not very 

at all difficult difficult 

2. How would you rank the clarity of your task? 


1 2 3 4 5 6 7 

not 

at all clear 


8 9 

very 
clear 


3. How closely does the task resemble what you do on a 
day-to-day basis in the fleet? 


1 23456789 

does not resembles 

resemble at all very closely 

4. Have you performed a similar task in the past? Y N 

a. If "YES" how long ago? (in years/months)_ 

5. Now that you have completed the task, can you think of any 
other factor (other variables, etc) that may have 
influenced you in making your decision? 


> 


4 


6. How clear were the instructions regarding the task? 

1 23456789 

Not clear Very 

at all clear 


J: 

7. How would you present the task 

differently?___ 
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8. How seriously did you take this task, generally? 


0 

not 

seriously 


) 10 
very 

seriously 


9. To what extent was information about the ship's actual 
position helpful in improving your own decision? 

1 2 3 4 5 6 7 8 9 

Not at all Very 

helpful helpful 


10. How easy was this system to use? 

1 2 3 4 5 6 

Not at all 
easy 


9 

Very 

easy 


-i 


4 - 
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MAP OF CARIBBEAN 

{National Geographic, 1988) 
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